Add read access to secret-DEPLOYMENT/edx-xqueue so the xqwatcher
service can retrieve the xqueue server URL and authentication
password needed by the ContainerGrader handler config.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Completely rewrite the xqwatcher Pulumi stack to deploy on Kubernetes
instead of EC2 Auto Scaling Groups with AppArmor/codejail.

Changes:
- Replace IAM instance profile + Vault AWS auth with OLEKSAuthBinding
  (IRSA + Vault K8s auth backend)
- Add OLVaultK8SSecret to sync grader handler config from Vault KV
  to a Kubernetes Secret via the Vault Secrets Operator CRD
- Add a ConfigMap for base poll settings and structured JSON logging
  to stdout (no log rotation in containers)
- Add RBAC Role + RoleBinding granting the xqwatcher service account
  permission to create/delete Kubernetes Jobs and read pod logs,
  required by ContainerGrader's kubernetes backend
- Create a Kubernetes Deployment with:
  - ghcr.io/mitodl/xqueue-watcher image
  - Security context (non-root, drop ALL capabilities)
  - Resource requests + memory limit
  - Liveness probe via python -c import xqueue_watcher
  - Topology spread for HA across nodes
  - Vault grader config + base config mounted into /xqwatcher/conf.d/
- Preserve vault.kv.SecretV2 write so grader config remains managed
  in Pulumi
- Export k8s_deployment_name and k8s_namespace

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove EC2-specific settings (consul:address, auto_scale, instance_type)
and add Kubernetes-specific settings for all stacks:

- xqwatcher:cluster — EKS cluster name (residential or applications)
- xqwatcher:namespace — target Kubernetes namespace
- xqwatcher:min_replicas — minimum pod count (maps from auto_scale.desired)
- xqwatcher:max_replicas — maximum pod count (maps from auto_scale.max)
- xqwatcher:docker_tag — container image tag (default: latest)

Cluster assignments:
- mitx, mitx-staging → residential cluster
- mitxonline → applications cluster

Namespace assignments follow xqueue convention:
- mitx → mitx-openedx
- mitxonline → mitxonline-openedx
- mitx-staging → mitx-staging-openedx

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

for more information, see https://pre-commit.ci

The manager CLI only accepts -d/--config_root; it auto-discovers
xqwatcher.json and logging.json from that directory. Remove the
non-existent --config and --logging-config flags.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…e tag

When the Concourse pipeline populates XQWATCHER_DOCKER_DIGEST, build
the image ref as mitodl/xqueue-watcher@sha256:... (immutable digest)
so Kubernetes always pulls exactly the image that was built and tested.
Fall back to :tag from stack config only when the digest is unavailable
(e.g. manual deploys). imagePullPolicy: Always is retained so new
digests are always pulled on rollout.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The uv virtualenv bin directory is not on PATH in the container, so
the 'xqueue-watcher' console script can't be found directly. Use
'uv run xqueue-watcher' to invoke it through uv's environment, which
correctly resolves the script installed in the project virtualenv.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

uv run without --no-sync attempts to sync the virtualenv at startup,
which fails in the container (no write access / network). Use
--no-sync to run the already-installed entrypoint as-is.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

VSO renders secret values via Go templates: {{ .Secrets.confd_json }}.
When confd_json is stored as a nested object, VSO renders a Go map
literal (map[...]) rather than valid JSON, causing a JSONDecodeError
at startup. Pre-serialize confd_json to a JSON string so the template
renders parseable JSON.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…llback

Match the keycloak pattern: require the digest env var so the image is
always pinned to an immutable digest. Remove the mutable :latest tag
fallback that allowed manual pulumi-up runs to silently deploy an
uncontrolled image. Also remove the unused xqwatcher:docker_tag config
key from all stack YAML files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…gh cache

When the SOPS secret's confd_json contains a ContainerGrader handler
whose KWARGS include an 'image' key, rewrite that value through
cached_image_uri() before writing to Vault. This means the SOPS secret
stores a plain DockerHub reference (e.g. mitodl/mit-600x-grader:latest)
and Pulumi transforms it to the ECR pull-through cache URI at deploy
time, keeping grading Jobs free from DockerHub rate limits.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The CodeQL 'Analyze (actions)' job (exit code 32) fails because the
extractor finds .github/workflows/*.yml and .github/actions/**/*.yml
but cannot process any of them. This is a known extractor-level issue
with CodeQL 2.24.x on Erk agent workflow patterns.

Excluding .github from CodeQL's path analysis silences the fatal error
while leaving Python and JavaScript/TypeScript scans unaffected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add src/ol_concourse/pipelines/open_edx/grader_images/ with three pipeline
definitions for building and publishing containerized course grader images
to private ECR.

base_image_pipeline.py:
  Builds grader_support/Dockerfile.base from the xqueue-watcher repo and
  pushes to both DockerHub (mitodl/xqueue-watcher-grader-base, public) and
  ECR (610119931565.dkr.ecr.us-east-1.amazonaws.com/mitodl/xqueue-watcher-
  grader-base, private). Triggered by changes to grader_support/ in the
  xqueue-watcher repo. The ECR push is the trigger source for downstream
  per-grader build pipelines.

build_pipeline.py:
  GraderPipelineConfig dataclass and grader_image_pipeline() factory for
  per-grader-repo build pipelines. Triggered by new commits to the grader
  repo OR a new base image digest in ECR. The Docker build receives
  GRADER_BASE_IMAGE=repo@sha256:... resolved at runtime via a sh wrapper
  around oci-build-task's build script (the only way to inject a
  file-derived BUILD_ARG in Concourse; params are static strings).
  Pushes to private ECR only. GRADER_PIPELINES list seeded with
  graders-mit-600x.

meta.py:
  Self-updating meta pipeline that creates and maintains the base image
  pipeline and one build pipeline per GRADER_PIPELINES entry. Triggered
  by changes to the grader_images/ pipeline code in ol-infrastructure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ines

- base_image_pipeline: use chore/migrate-to-uv-and-k8s-container-grader
  branch of xqueue-watcher (where Dockerfile.base updates live)
- build_pipeline: track feat/containerized-grader for graders-mit-600x
- Fix E501 in both files: split long strings to stay within 88-char limit

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The CONTEXT was grader_support/ which caused the COPY grader_support/
instruction in Dockerfile.base to fail (no nested grader_support/ inside
the context). Use the repo root as CONTEXT so the COPY can locate the
directory relative to it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…images

Add ensure_ecr_task() helper to ol_concourse/lib/containers.py (mirrors the
pattern used in the dagster docker_pulumi_pipeline). The task runs the AWS
CLI to check for the ECR repository and creates it if missing, so the first
pipeline run does not fail on a missing registry.

Apply to both grader image pipelines:
- base_image_pipeline: ensures mitodl/xqueue-watcher-grader-base exists
  before pushing to ECR
- build_pipeline: ensures the per-grader ECR repo (config.ecr_repo_name)
  exists before pushing the course grader image

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When ecr_region is set, the registry-image resource automatically
constructs the full ECR URI as {account}.dkr.ecr.{region}.amazonaws.com/{repository}.
Passing the full URI in image_repository caused the hostname to be doubled
in API calls, resulting in NAME_UNKNOWN errors.

- Remove ecr_image_uri property from GraderPipelineConfig
- Fix grader_base_ecr_repo default to use repo-name-only string
- Change registry_image(image_repository=config.ecr_image_uri) to
  registry_image(image_repository=config.ecr_repo_name)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The grader-images-pipeline-code git resource was tracking 'main', but
the pipeline files don't exist on main yet. Switch to the feature branch
until this work is merged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

infrastructure/github has no generic SSH key. The correct key for
cloning private mitodl repos from the infrastructure Concourse team
is odlbot_private_ssh_key in infrastructure/open_api_clients.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…onfig + SERVER_REF

Queue configs (CONNECTIONS, HANDLERS, ContainerGrader KWARGS) are now
stored as plaintext in Pulumi stack YAML files under xqwatcher:queues.
The xqueue server URL is stored under xqwatcher:xqueue_server_url.

SERVER_REF is injected at deploy time so xqueue-watcher resolves
credentials at runtime from xqueue_servers.json, which is mounted from
a Vault-synced Kubernetes Secret.  The secret is sourced from the same
secret-{env_prefix}/edx-xqueue Vault KV path already used by the xqueue
and edxapp deployments (xqwatcher_password field), eliminating the
separate xqwatcher-specific KV mount and SOPS secrets files.

Changes:
- __main__.py: remove SOPS read, vault.kv.SecretV2, vault_mount_stack
  StackReference, and XQWATCHER_HTTP_BASIC_AUTH env var; read queues
  config from Pulumi config; inject SERVER_REF into each queue entry;
  move grader_config.json into ConfigMap; add xqueue_servers.json
  Vault-synced secret from secret-{env_prefix}/edx-xqueue; update
  Deployment volumes/mounts accordingly
- xqwatcher_server_policy.hcl: remove secret-xqwatcher/* path
- All 9 stack YAML files: add xqueue_server_url and queues config

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Unpin grader_images meta pipeline from feature branch; track main
- Unpin xqueue-watcher base image source from dev branch; track main
- Unpin graders-mit-600x grader repo from feature branch; track main
- Fix base_image_pipeline.py docstring: downstream pipelines trigger off
  the DockerHub push, not the ECR push
- Add xqwatcher:docker_tag config fallback for XQWATCHER_DOCKER_DIGEST
  so pulumi up can run without the env var set (matches xqueue pattern)
- Remove env vars that duplicate xqwatcher.json ConfigMap values
  (POLL_TIME, REQUESTS_TIMEOUT, POLL_INTERVAL, FOLLOW_CLIENT_REDIRECTS);
  keep only LOGIN_POLL_INTERVAL and GRADER_* which are not in the ConfigMap
- Update PR description: image is on DockerHub, not GHCR

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add an "edxorg" entry to the xqueue_servers.json Vault template so that
queues using SERVER_REF "edxorg" resolve credentials for
https://xqueue.edx.org. The template variables edxorg_xqueue_username
and edxorg_xqueue_password must be added to the existing edx-xqueue
Vault KV secret.

Update the queue config loop to use setdefault so that queues can
declare their own SERVER_REF in the Pulumi stack config rather than
always being assigned "default".

Set SERVER_REF: edxorg on the edxorg-686x queue in the mitxonline
production stack config.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ofile

Accommodates changes from xqueue-watcher commit 15fdd86
(security: harden containergrader and XQueue client):

- Expose XQWATCHER_VERIFY_TLS via xqwatcher:verify_tls Pulumi config
  (default "true"; set "false" only for dev envs with self-signed certs).
- Expose XQWATCHER_SUBMISSION_SIZE_LIMIT via xqwatcher:submission_size_limit
  Pulumi config (default 1 MB, matching containergrader default).
- Add RuntimeDefault seccomp profile to the xqwatcher pod's
  PodSecurityContextArgs, mirroring the profile now applied to grading
  Jobs in containergrader.py for defence-in-depth consistency.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Add a HorizontalPodAutoscaler (autoscaling/v2) targeting the xqwatcher
Deployment, scaling on:
- CPU: 60% average utilization
- Memory: 80% average utilization

Scale-up is aggressive (up to 100% more pods per minute, 60s
stabilization) to handle submission bursts; scale-down is conservative
(≤25% reduction per minute, 5-minute stabilization) to avoid thrashing.

Min/max replica bounds are configurable via xqwatcher:min_replicas and
xqwatcher:max_replicas stack config (defaults: 1 and 5).

The Deployment gains ignore_changes=["spec.replicas"] so Pulumi does
not revert the replica count that the HPA manages between stack updates.

Exports k8s_hpa_name for stack consumers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The original design embedded edxorg credentials in the same
VaultStaticSecret as the MIT-hosted xqueue server, referencing
edxorg_xqueue_username / edxorg_xqueue_password keys that do not exist
in secret-<env>/edx-xqueue (which only holds edxapp_password and
xqwatcher_password).

Instead, create a fully independent Deployment per xqueue server:

- xqwatcher (default): watches queues targeting the MIT-hosted xqueue.
  Reads credentials from secret-<env>/edx-xqueue.  Only queues with
  SERVER_REF="default" (or no SERVER_REF) are included in its ConfigMap.

- xqwatcher-edxorg (optional): watches queues with SERVER_REF="edxorg".
  Reads credentials from a separate secret-<env>/edxorg-xqueue Vault
  path, created only when xqwatcher:edxorg_xqueue_enabled=true.

Each Deployment has its own ConfigMap (scoped to its own queue subset),
VaultStaticSecret, and HPA; they share the xqwatcher ServiceAccount and
RBAC Role since both need identical permissions to manage grading Jobs.
The Vault policy gains a read grant for secret-DEPLOYMENT/edxorg-xqueue.

This eliminates the need for a file-merge initContainer and gives each
server integration independent observability, scaling, and secret access.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>